Automating the Math

---

Transcribing with MathJax and SRE

Background: Initial Problem

• No reliable, fast, and inexpensive way of producing Braille versions of mathematics texts.
• Several tools will produce convincing-looking, but incorrect output.
• Situation is particular bad for graduate texts
• And getting worse for research papers, etc.

Overview

• Math document formats and their accessibility
  • PreTeXt and its accessibilty
• Accessible Math with MathJax/SRE
• Creating Tactile Hardcopies
  • From PreTeXt to Tactile
  • Nemeth with MathJax and Speech Rule Engine
• A word on graphics with screen reading

Document formats with math support

• LaTeX: to CTAN and beyond!
• Microsoft Word: OOXML, "UnicodeMath", LaTeX (limited, input only)
• Libre Writer: ODT Math, custom ascii
• Apple Pages: blahtex, mathml
• lightweight markup: Markdown (varying, by implementations/extensions), asciidoc (asciimath), wikitext (TeX-like)

Accessibility of these formats

• Word formats
  • Raw: Pretty good accessibility in editor, Math depends on implementation/screen reader/formula editor!
  • As PDF: Good structural accessibility, but no math
• LaTeX needs to be compiled into PDF
  • Although LaTeX provides great structure that is lost in PDF
  • Math should be good but is lost too!
• Markup like Markdown/ HTML
  • Good structural accessibility
  • Math is accessible if handled right

Best option is often to move content to the web

Conversion towards web formats

• pandoc - bi-directional "swiss army knife"
  • markdown, TeX, docx, and many many more
• LaTeX
  • tex4ht (make4ht)
  • plastex
  • latexml
  • and many more...
• PDF
  • pdf2htmlex
  • PDF.js

Keep as much of the source as possible: do not replace the LaTeX

Why is LaTeX important

• LaTeX notation is lingua franca in Math
• Also usable in informal communication like email
• Can be used on the web (github, stackoverflow, etc.)
• Every new Math editor ultimately allows LaTeX as input syntax
• LaTeX for the visually impaired
  • Some European countries teach LaTeX to VI stduents from grade 1
  • Little additional effort to learn
  • Easy to communicate with non-specailist teachers and sighted peers
  • Translation of code to 8-dot LaTeX Braille

Creating Tactile Hardcopies

Translate textbooks automatically into Braille from source

• Little support in for most formats
• LaTeX package do not produce quality results

What do we do?

• Sources in PreTeXt
• Text into literary Braille with LibLouis
• Math into Nemeth with MathJax and Speech Rule Engine
• Graphics into tactile graphics via a multistep pipeline

What is PreTeXt?

PreTeXt is a an uncomplicated XML vocabulary for open source textbooks, monographs, and research articles.

Aims to bring together the good parts of HTML/Word with LaTeX.

• Structure in XML
• Mathematics in LaTeX
• Graphics mainly in LaTeX
• Over 100 individual projects:
  • mainly Math, some CS and engineering
  • one music theory, one university writing handbook

In our context: provides free and well curated content

PreTeXt and Accessibility

• One source, many outputs:
  • print, PDF, web, EPUB, Jupyter Notebooks, ...
• HTML output is accessible
  • Mathematics is in LaTeX
  • Rendered with MathJax
  • Made accessible using Speech Rule Engine

What is MathJax?

• MathJax is an open-source JavaScript library for rendering Mathematics in all browsers
• Can take {\LaTeX}, AsciiMath, and MathML as input
• Generates browser output, e.g. HTML/CSS, SVG
• MathJax is the de facto rendering solution of most Mathematics on the web
• Standard Maths rendering solution for many publishers
• Around for over 10 years, lastest version 3.1 released Aug 2020

In our context: translation of LaTeX to Nemeth Braille

Speech Rule Engine

• Javascript library for translating Math into Speech available at github
• Only Math speech solution in JavaScript: Runs in browser, command line, as node module.
• Speech solution for: ChromeVox, MathJax, EquatIO, MathLive, ...
• Based on semantic interpretations via pattern matching techniques
  • Ignores any poor syntax (or at least tries to)
  • Math markup (e.g., LaTeX, MathML) are not very expressive, sometimes bloated

SRE in MathJax

$$x = {-b \pm \sqrt{b^2-4ac} \over 2a}$$

• SRE is pulled into MathJax
• Speech (or Braille) with
  • multiple reading rules and preference,
  • different languages
  • Play Area for speech
• Interactive Exploration
  • Highlighting, magnification, etc.
• All this is done using a semantic interpretation

Generate Semantics

• Consider $ax^2+bx+c=0$. SRE rewrites this into semantic term structure

    ax^2+bx+c=0

<math>
  <mi>a</mi>
  <msup>
    <mi>x</mi>
    <mn>2</mn>
  </msup>
  <mo>+</mo>
  <mi>b</mi>
  <mi>x</mi>
  <mo>+</mo>
  <mi>c</mi>
  <mo>=</mo>
  <mn>0</mn>
</math>

Generate Speech

$$x = {-b \pm \sqrt{b^2-4ac} \over 2a}$$

• Math voiced via a screen reader
• Autovoicing

Speech

Play Area for speech

• Multiple reading rules and preferences that can be swapped
  • MathSpeak, ClearSpeak, (ChromeVox)
  • Summarisation, Navigation info
• 12 different locales
  • English, German, Spanish, French, Italian, ...
• Tactile formats
  • 8-dot LaTeX Braille (German, Slovenian, ...)
  • Nemeth (linear)
  • Nemeth 2D

Nemeth in SRE

Sponsored by Big Ten Academic Alliance

• Once semantics is embedded speech and other alternative formats can be easily generated
• Tactile output, Abstraction, Summaries $$ \vec{\nabla} \times \vec{F} = \left( \frac{\partial F_z}{\partial y} - \frac{\partial F_y}{\partial z} \right) \mathbf{i} + \left( \frac{\partial F_x}{\partial z} - \frac{\partial F_z}{\partial x} \right) \mathbf{j} + \left( \frac{\partial F_y}{\partial x} - \frac{\partial F_x}{\partial y} \right) \mathbf{k} $$

MathJax on the Server

• work on the server as easily as in the browser
• works in non-standard DOMs (e.g., markdown or PreTeXt)
• separable pipeline for
  • finding
  • converting
  • annotating
  • rendering math
• E.g., LaTeX in a document can be translated into SVG with speech or Braille annotation.

2D in Nemeth

• Nemeth for embossing requires
  • Large fractions
  • Matrices, vectors
  • Equation systems
  • Cayley tables
• SRE can generate 2D output
• Uses same rules as for Braille display output during screen reading
• Applies abstract styling rules
• Some sample output

Quality Assurance

• We developed a Webapp for Nemeth Translation
• Basic idea:
  • Proof read and correct automatically translated math
  • Easy reference to context via links source
  • Selection of semantically unique samples vs full list of expressions

Some Challenging Examples

Nemeth is challenging to get right automatically, but sometimes automation is better:

• Subtlety of spaces:
  • $44,352,000$ is a single number and transcribes as ⠼⠲⠲⠒⠢⠆⠴⠴⠴
  • $(0110,1110,0110)$ is not a single number but a vector, hence ⠷⠼⠴⠂⠂⠴⠀⠼⠂⠂⠂⠴⠀⠼⠴⠂⠂⠴⠾
  • $(n, E) = (451{,}231)$ The right hand side is not a single number ⠷⠝⠠⠀⠠⠑⠾⠀⠨⠅⠀⠷⠲⠢⠂⠠⠀⠆⠒⠂⠾
• Context helps $$PAP^{-1} = B$$ is not an abbreviation: ⠠⠏⠠⠁⠠⠏⠘⠤⠂⠀⠨⠅⠀⠠⠃

Voicing Complex Content

$$ \require{bussproofs} \begin{prooftree} \AxiomC{} \RightLabel{$Hyp^{1}$} \UnaryInfC{$P$} \AXC{$P\to Q$} \RL{$\to_E$} \BIC{$Q^2$} \AXC{$Q\to R$} \RL{$\to_E$} \BIC{$R$} \AXC{$Q$} \RL{Rit$^2$} \UIC{$Q$} \RL{$\wedge_I$} \BIC{$Q\wedge R$} \RL{$\to_I^1$} \UIC{$P\to Q\wedge R$} \end{prooftree} $$

• Try to express this in MathML

What about Diagrams?

• Automated generation of tactile graphics is challenging
• Advanced, bespoke content
• Generated with LaTeX packages
• Many Standard techniques do not work

Some examples from Abstract Algebra

• Roots of Unity
• Fairly easy relations
• Large graphics with interspersed math
• Lattices, Category theory, etc.

Processing Labels

• Easy: Separate Text vs Math
  • PreTeXt already does this for us: Text vs LaTeX
  • Translate Text with Liblouis
  • Translate Math with SRE
• Hard: Fitting tactile labels into graphic
• Hard: Understanding graphical content

Processing Labels

• Easy: Separate Text vs Math
  • PreTeXt already does the for us: Text vs LaTeX
  • Translate Text with Liblouis
  • Translate Math with SRE
• Hard: Fitting labels into graphic
  • Braille uses different space than text
  • Labels are measured
  • Bounding boxes are created and aligned

Example: Latex to Tactile

Source: Tom Judson, Abstract Algebra: Theory and Application

Workflow Example

• Creating Braille bounding boxes in the diagram

Workflow Example

• Fitting ASCII Braille into bounding boxes

Workflow Example

• Full rendering in Unicode Braille Font

Diagrams for Screenreading

• Once the graphics are understood we can do much more
• Diagrams can be handled via screen readers as well
• Goal is to have them all similar to our chemistry at progacc.com/chemistry

Talking to the Embosser

• The entire image is measured and scaled as much as possible to fit onto a page
• Good-looking image on the screen frequently does not look good embossed
• Result is usually suboptimal
  • Scaling of graphics should not apply to Braille
  • Embosser do not work well with SVG
• Conversion to PDF necessary
• Fonts need to be explicitly un-embedded

Example: Poorly embossed

• Diagram in PDF with fonts

Example: Legibly embossed

• Unembedded fonts

Example: Best solution

• Smoothing applied

Some Conclusions

• Making Math accessible on the web is quite straightforward
  • Make sure to work from sources
  • Do not replace LaTeX with MathML!
• MathJax v4 is about to be released
• Currently Nemeth Braille:
  • Adding other variants, e.g. UEB
  • Definitely 8-dot LaTeX Braille
• Missing parts:
  • Intelligent Linebreaking
  • Form factors and Reflow
• Diagrams are really exciting... More tomorrow!